{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reuters Multilabel NLP\n",
    "\n",
    "In this notebook, I'll explore the Reuters corpus from NLTK using a few different NLP techniques. The corpus is 'multilabel' (which is different from standard 'multiclass'), meaning each article (I'll look at ~1,000) is associated with multiple categories, or tags. This is similar to a blog or website that may have a post classified as 'technology', 'machine learning' and 'natural language processing' simultaneously.\n",
    "\n",
    "I'll use both Bag of Words and tfidf to create feature spaces to run through supervised and unsupervised models. Because metrics are difficult for unsupervised multilabel models, I've designed a custom function to evaluate accuracy for training and test sets.\n",
    "\n",
    "Happy reading!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:04.393582Z",
     "start_time": "2018-03-02T17:50:02.737093Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import re\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set_style('white')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examine the Data\n",
    "\n",
    "First, let's load up the data and get a feel for how it's organized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:04.969631Z",
     "start_time": "2018-03-02T17:50:04.396383Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let's use articles from the Reuters Corpus.\n",
    "from nltk.corpus import reuters, stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:06.535249Z",
     "start_time": "2018-03-02T17:50:04.972584Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tin', 'coffee', 'gold', 'yen', 'rubber', 'palm-oil', 'lin-oil', 'rape-oil', 'wheat', 'soy-oil', 'nickel', 'cocoa', 'sugar', 'nzdlr', 'sorghum']\n"
     ]
    }
   ],
   "source": [
    "# We'll choose some categories at random and use those for the rest of our analysis.\n",
    "np.random.seed(3)\n",
    "categories = list(np.random.choice(reuters.categories(), 15, replace=False))\n",
    "print(categories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:06.550074Z",
     "start_time": "2018-03-02T17:50:06.540222Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(283, 'wheat'),\n",
       " (162, 'sugar'),\n",
       " (139, 'coffee'),\n",
       " (124, 'gold'),\n",
       " (73, 'cocoa'),\n",
       " (59, 'yen'),\n",
       " (49, 'rubber'),\n",
       " (40, 'palm-oil'),\n",
       " (34, 'sorghum'),\n",
       " (30, 'tin'),\n",
       " (25, 'soy-oil'),\n",
       " (9, 'nickel'),\n",
       " (8, 'rape-oil'),\n",
       " (4, 'nzdlr'),\n",
       " (2, 'lin-oil')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check that our categories have relatively the same order of magnitude of articles associated with them.\n",
    "sorted([(len(reuters.fileids(x)), x) for x in categories], reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:08.939526Z",
     "start_time": "2018-03-02T17:50:06.554278Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of articles: 962\n",
      "['test/14832', 'test/14833', 'test/14840', 'test/14841', 'test/14842']\n",
      "Number of total words: 227749\n"
     ]
    }
   ],
   "source": [
    "# See how many articles are attached to those categories, the filenames of a few, and total word count.\n",
    "print('Number of articles:', len(reuters.fileids(categories)))\n",
    "fileids = reuters.fileids(categories)\n",
    "print(fileids[:5])\n",
    "print('Number of total words:', len(reuters.words(fileids)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in the Reuters corpus, categories are overlapping and each article can be tagged with multiple categories. This may add some complexity to our analysis, but it's also more representative of a real-world NLP project where specific web pages may have multiple topics associated with them.\n",
    "\n",
    "Although we won't be analyzing them, it is a good idea to see if there are other categories that have been pulled into our data set. These would be from additional tags shared with the articles, but that we didn't explicitly select for (for instance, all of our 'wheat' articles also are tagged 'grain')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:08.962475Z",
     "start_time": "2018-03-02T17:50:08.943457Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[311, 'grain'],\n",
       " [283, 'wheat'],\n",
       " [162, 'sugar'],\n",
       " [139, 'coffee'],\n",
       " [124, 'gold'],\n",
       " [104, 'corn'],\n",
       " [73, 'cocoa'],\n",
       " [68, 'oilseed'],\n",
       " [67, 'veg-oil'],\n",
       " [59, 'yen'],\n",
       " [56, 'money-fx'],\n",
       " [51, 'soybean'],\n",
       " [49, 'rubber'],\n",
       " [40, 'palm-oil'],\n",
       " [36, 'dlr']]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We may also be pulling in a lot of other categories in this process because articles can have multiple associations.\n",
    "# Let's see all the categories we're getting.\n",
    "all_categories = reuters.categories(fileids)\n",
    "category_counts = [[len(set(reuters.fileids(x)).intersection(fileids)), x] for x in all_categories]\n",
    "sorted(category_counts, reverse=True)[:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, we've got a few common categories that we didn't choose explicitly, but which turned out to be popular across the articles that were selected. For instance, 'grain' wasn't in the initial list, but 'wheat' and 'sorghum' are both grains, so it is popular. 'Corn' and 'veg-oil' as well. This should be fine -- it means we may end up clustering by some of these more general categories during unsupervised learning.\n",
    "\n",
    "Let's see how the articles are stored in the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:11.101576Z",
     "start_time": "2018-03-02T17:50:11.074378Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw:\n",
      " THAI TRADE DEFICIT WIDENS IN FIRST QUARTER\n",
      "  Thailand's trade deficit widened to 4.5\n",
      "  billion baht in the first quarter of 1987 from 2.1 billion a\n",
      "  year ago, the Business Economics Department said.\n",
      "      It said Janunary/March imports rose to 65.1 \n",
      "\n",
      "Words:\n",
      " ['THAI', 'TRADE', 'DEFICIT', 'WIDENS', 'IN', 'FIRST', 'QUARTER', 'Thailand', \"'\", 's', 'trade', 'deficit', 'widened', 'to', '4', '.', '5', 'billion', 'baht', 'in', 'the', 'first', 'quarter', 'of', '1987', 'from', '2', '.', '1', 'billion', 'a', 'year', 'ago', ',', 'the', 'Business', 'Economics', 'Department', 'said', '.', 'It', 'said', 'Janunary', '/', 'March', 'imports', 'rose', 'to', '65', '.']\n",
      "\n",
      "\n",
      "Categories:\n",
      " ['corn', 'grain', 'rice', 'rubber', 'sugar', 'tin', 'trade']\n"
     ]
    }
   ],
   "source": [
    "# There are three ways we can look at each file: Full text, words, and categories.\n",
    "# We'll use 'words' for BoW, full text for Tfidf, and categories for classification in both.\n",
    "print('Raw:\\n', reuters.raw('test/14832')[:250])\n",
    "print('\\nWords:\\n', reuters.words('test/14832')[:50])\n",
    "print('\\n\\nCategories:\\n', reuters.categories('test/14832'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the Data\n",
    "\n",
    "Because we're already given a method for accessing individual words, we'll have a bit less data cleaning to do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:14.910301Z",
     "start_time": "2018-03-02T17:50:12.902952Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: SUBROTO SAYS INDONESIA SUPPORTS TIN PACT EXTENSION\n",
      "  Mines and Energy Minister Subroto\n",
      "  confirmed Indonesian support for an extension of the sixth\n",
      "  International Tin Agreement (ITA), but said a new  \n",
      "\n",
      "After: subroto says indonesia supports tin pact extension mines and energy minister subroto confirmed indonesian support for an extension of the sixth international tin agreement (ita), but said a new pact w \n",
      "\n",
      "[['corn', 'grain', 'rice', 'rubber', 'sugar', 'tin', 'trade'], ['palm-oil', 'veg-oil'], ['coffee', 'lumber', 'palm-oil', 'rubber', 'veg-oil'], ['grain', 'wheat'], ['gold']]\n"
     ]
    }
   ],
   "source": [
    "# Create a list with each article as an element.\n",
    "articles = [reuters.raw(x) for x in fileids]\n",
    "print('Before:', articles[5][:200], '\\n')\n",
    "\n",
    "# Let's make them all lowercase and remove white space.\n",
    "articles = [' '.join(x.split()).lower() for x in articles]\n",
    "print('After:', articles[5][:200], '\\n')\n",
    "\n",
    "# And grabbing categories by article.\n",
    "categories_by_article = [reuters.categories(x) for x in fileids]\n",
    "print(categories_by_article[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That should be enough for the raw text, let's just touch up the 'Words' lists by making lowercase and removing punctuation, as we won't be using that for BoW. Instead of grabbing all the punctuation separately, I'll just get rid of all words length 2 or less and all stopwords. This will also deal with some common words and numbers and make our data a little easier to wrangle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:20.920686Z",
     "start_time": "2018-03-02T17:50:14.913438Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before: ['SUBROTO', 'SAYS', 'INDONESIA', 'SUPPORTS', 'TIN', 'PACT', 'EXTENSION', 'Mines', 'and', 'Energy', 'Minister', 'Subroto', 'confirmed', 'Indonesian', 'support', 'for', 'an', 'extension', 'of', 'the', 'sixth', 'International', 'Tin', 'Agreement', '(']\n",
      "Total Words: 227749 \n",
      "\n",
      "After: ['subroto', 'says', 'indonesia', 'supports', 'tin', 'pact', 'extension', 'mines', 'energy', 'minister', 'subroto', 'confirmed', 'indonesian', 'support', 'extension', 'sixth', 'international', 'tin', 'agreement', 'ita', 'said', 'new', 'pact', 'necessary', 'asked']\n",
      "Total Words: 119008\n"
     ]
    }
   ],
   "source": [
    "# Make a list of all the words in each article.\n",
    "words_by_article = [reuters.words(x) for x in fileids]\n",
    "print('Before:', words_by_article[5][:25])\n",
    "print('Total Words:', sum([len(x) for x in words_by_article]), '\\n')\n",
    "\n",
    "# A set of stopwords will be faster to search through.\n",
    "stopword_set = set(stopwords.words('english'))\n",
    "for i, words in enumerate(words_by_article):\n",
    "    words_by_article[i] = [x.lower() for x in words if (len(x) > 2) and not (x in stopword_set)]\n",
    "print('After:', words_by_article[5][:25])\n",
    "print('Total Words:', sum([len(x) for x in words_by_article]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nice, we've cut out nearly half the words and punctuation from our data set that were unlikely to be useful. This should make things go faster from here on out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag of Words\n",
    "\n",
    "Are we going to use lemmas, etc for this analysis? It's a bit slow and we've already got a nice clean set of words. Let's see how we do without getting into spaCy first, and add it if it seems useful. (Note: it turned out not to be necessary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up BoW\n",
    "\n",
    "First we'll build our Bag of Words by finding the most common words and counting them up in each article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:26.212242Z",
     "start_time": "2018-03-02T17:50:26.181870Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import itertools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:26.818924Z",
     "start_time": "2018-03-02T17:50:26.782912Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('said', 3481), ('the', 1749), ('tonnes', 1535), ('mln', 1331), ('000', 1176), ('wheat', 961), ('year', 814), ('sugar', 731), ('pct', 713), ('would', 682), ('coffee', 650), ('last', 616), ('1986', 578), ('export', 539), ('market', 539), ('dlrs', 537), ('prices', 527), ('gold', 518), ('new', 512), ('price', 442), ('trade', 425), ('may', 408), ('production', 391), ('cocoa', 376), ('producers', 363)]\n"
     ]
    }
   ],
   "source": [
    "# Put all the strings into a single list.\n",
    "allwords = [word for article in words_by_article for word in article]\n",
    "\n",
    "# Count 'em up.\n",
    "words_counted = Counter(allwords)\n",
    "\n",
    "# Which words are most common?\n",
    "print(words_counted.most_common(25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these words aren't great to have in our set. Let's remove some by hand, and then remove words that occur less than 3 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:28.413555Z",
     "start_time": "2018-03-02T17:50:28.399294Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('tonnes', 1535), ('mln', 1331), ('wheat', 961), ('sugar', 731), ('would', 682), ('coffee', 650), ('last', 616), ('1986', 578), ('export', 539), ('market', 539), ('dlrs', 537), ('prices', 527), ('gold', 518), ('new', 512), ('price', 442), ('trade', 425), ('production', 391), ('cocoa', 376), ('producers', 363), ('nil', 361), ('1987', 358), ('week', 351), ('one', 347), ('per', 347), ('stock', 346)]\n",
      "Unique words: 9750\n"
     ]
    }
   ],
   "source": [
    "# Make a list of words to ignore, then cycle through the counter to remove them.\n",
    "ignore = ['said', 'the', '000', 'year', 'pct', 'may']\n",
    "for word in list(words_counted):\n",
    "    if word in ignore:\n",
    "        del words_counted[word]\n",
    "print(words_counted.most_common(25))\n",
    "print('Unique words:', len(list(words_counted)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:29.736519Z",
     "start_time": "2018-03-02T17:50:29.580510Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Least common words: [('lechin', 5), ('arbitration', 5), ('awf', 5), ('martell', 5), ('citicorp', 5)]\n",
      "Unique words: 3333\n"
     ]
    }
   ],
   "source": [
    "# We'll use itertools to delete all items that are not dropped by this lambda function, \n",
    "# which is dropping everything that occurs at least 5 times.\n",
    "for key, count in itertools.dropwhile(lambda key_count: key_count[1] >= 5, words_counted.most_common()):\n",
    "    del words_counted[key]\n",
    "print('Least common words:', words_counted.most_common()[::-1][:5])    \n",
    "print('Unique words:', len(list(words_counted)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "That's a bit better. Let's go ahead and make of DataFrame for modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:50:33.130057Z",
     "start_time": "2018-03-02T17:50:31.577683Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_text</th>\n",
       "      <th>categories</th>\n",
       "      <th>thai</th>\n",
       "      <th>trade</th>\n",
       "      <th>deficit</th>\n",
       "      <th>first</th>\n",
       "      <th>quarter</th>\n",
       "      <th>thailand</th>\n",
       "      <th>billion</th>\n",
       "      <th>1987</th>\n",
       "      <th>...</th>\n",
       "      <th>nogales</th>\n",
       "      <th>awf</th>\n",
       "      <th>amselco</th>\n",
       "      <th>amt</th>\n",
       "      <th>arbitration</th>\n",
       "      <th>lechin</th>\n",
       "      <th>otsuki</th>\n",
       "      <th>miti</th>\n",
       "      <th>iwc</th>\n",
       "      <th>witte</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>thai trade deficit widens in first quarter tha...</td>\n",
       "      <td>[corn, grain, rice, rubber, sugar, tin, trade]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>indonesia sees cpo price rising sharply indone...</td>\n",
       "      <td>[palm-oil, veg-oil]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>indonesian commodity exchange may expand the i...</td>\n",
       "      <td>[coffee, lumber, palm-oil, rubber, veg-oil]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 3335 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        article_text  \\\n",
       "0  thai trade deficit widens in first quarter tha...   \n",
       "1  indonesia sees cpo price rising sharply indone...   \n",
       "2  indonesian commodity exchange may expand the i...   \n",
       "\n",
       "                                       categories  thai  trade  deficit  \\\n",
       "0  [corn, grain, rice, rubber, sugar, tin, trade]     0      0        0   \n",
       "1                             [palm-oil, veg-oil]     0      0        0   \n",
       "2     [coffee, lumber, palm-oil, rubber, veg-oil]     0      0        0   \n",
       "\n",
       "   first  quarter  thailand  billion  1987  ...    nogales  awf  amselco  amt  \\\n",
       "0      0        0         0        0     0  ...          0    0        0    0   \n",
       "1      0        0         0        0     0  ...          0    0        0    0   \n",
       "2      0        0         0        0     0  ...          0    0        0    0   \n",
       "\n",
       "   arbitration  lechin  otsuki  miti  iwc  witte  \n",
       "0            0       0       0     0    0      0  \n",
       "1            0       0       0     0    0      0  \n",
       "2            0       0       0     0    0      0  \n",
       "\n",
       "[3 rows x 3335 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a DataFrame, include article text, categories, and features for all the words in our list.\n",
    "df_bow = pd.DataFrame(columns=['article_text', 'categories'] + list(words_counted))\n",
    "df_bow['article_text'] = articles\n",
    "df_bow['categories'] = categories_by_article\n",
    "df_bow.fillna(0, inplace=True)\n",
    "df_bow.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we'll populate the word counts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:01.633664Z",
     "start_time": "2018-03-02T17:50:40.130433Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_text</th>\n",
       "      <th>categories</th>\n",
       "      <th>thai</th>\n",
       "      <th>trade</th>\n",
       "      <th>deficit</th>\n",
       "      <th>first</th>\n",
       "      <th>quarter</th>\n",
       "      <th>thailand</th>\n",
       "      <th>billion</th>\n",
       "      <th>1987</th>\n",
       "      <th>...</th>\n",
       "      <th>nogales</th>\n",
       "      <th>awf</th>\n",
       "      <th>amselco</th>\n",
       "      <th>amt</th>\n",
       "      <th>arbitration</th>\n",
       "      <th>lechin</th>\n",
       "      <th>otsuki</th>\n",
       "      <th>miti</th>\n",
       "      <th>iwc</th>\n",
       "      <th>witte</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>thai trade deficit widens in first quarter tha...</td>\n",
       "      <td>[corn, grain, rice, rubber, sugar, tin, trade]</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>indonesia sees cpo price rising sharply indone...</td>\n",
       "      <td>[palm-oil, veg-oil]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>indonesian commodity exchange may expand the i...</td>\n",
       "      <td>[coffee, lumber, palm-oil, rubber, veg-oil]</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sri lanka gets usda approval for wheat price f...</td>\n",
       "      <td>[grain, wheat]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>western mining to open new gold mine in austra...</td>\n",
       "      <td>[gold]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 3335 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        article_text  \\\n",
       "0  thai trade deficit widens in first quarter tha...   \n",
       "1  indonesia sees cpo price rising sharply indone...   \n",
       "2  indonesian commodity exchange may expand the i...   \n",
       "3  sri lanka gets usda approval for wheat price f...   \n",
       "4  western mining to open new gold mine in austra...   \n",
       "\n",
       "                                       categories  thai  trade  deficit  \\\n",
       "0  [corn, grain, rice, rubber, sugar, tin, trade]     1      2        2   \n",
       "1                             [palm-oil, veg-oil]     0      0        0   \n",
       "2     [coffee, lumber, palm-oil, rubber, veg-oil]     0      3        0   \n",
       "3                                  [grain, wheat]     0      0        0   \n",
       "4                                          [gold]     0      0        0   \n",
       "\n",
       "   first  quarter  thailand  billion  1987  ...    nogales  awf  amselco  amt  \\\n",
       "0      4        4         2        6     1  ...          0    0        0    0   \n",
       "1      0        0         0        0     0  ...          0    0        0    0   \n",
       "2      1        0         0        1     1  ...          0    0        0    0   \n",
       "3      0        0         0        0     0  ...          0    0        0    0   \n",
       "4      1        0         0        0     0  ...          0    0        0    0   \n",
       "\n",
       "   arbitration  lechin  otsuki  miti  iwc  witte  \n",
       "0            0       0       0     0    0      0  \n",
       "1            0       0       0     0    0      0  \n",
       "2            0       0       0     0    0      0  \n",
       "3            0       0       0     0    0      0  \n",
       "4            0       0       0     0    0      0  \n",
       "\n",
       "[5 rows x 3335 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make a set of the feature names for faster searching.\n",
    "features_set = set(words_counted)\n",
    "\n",
    "# Iterate through the words in each article to add values to our DataFrame.\n",
    "for i, article in enumerate(words_by_article):\n",
    "    counter = Counter(article)\n",
    "    for key, count in counter.most_common():\n",
    "        if key in features_set:\n",
    "            df_bow.loc[i, key] = count\n",
    "            \n",
    "df_bow.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:01.645602Z",
     "start_time": "2018-03-02T17:51:01.636595Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Trade\" added up over the DataFrame: 425\n",
      "\"Trade\" from our original \"words_counted\" counter: 425\n"
     ]
    }
   ],
   "source": [
    "# Sanity check -- did we get everything counted?\n",
    "print('\"Trade\" added up over the DataFrame:', df_bow['trade'].sum())\n",
    "print('\"Trade\" from our original \"words_counted\" counter:', words_counted['trade'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Supervised BoW Model\n",
    "\n",
    "We're ready to begin modeling! We still have to create our X and y matrices and split the data into training and testing. We'll have to transform our outcome variable, y, into a binary array of multilabel outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:08.270554Z",
     "start_time": "2018-03-02T17:51:08.244385Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df shape: (962, 3335)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_text</th>\n",
       "      <th>categories</th>\n",
       "      <th>thai</th>\n",
       "      <th>trade</th>\n",
       "      <th>deficit</th>\n",
       "      <th>first</th>\n",
       "      <th>quarter</th>\n",
       "      <th>thailand</th>\n",
       "      <th>billion</th>\n",
       "      <th>1987</th>\n",
       "      <th>...</th>\n",
       "      <th>nogales</th>\n",
       "      <th>awf</th>\n",
       "      <th>amselco</th>\n",
       "      <th>amt</th>\n",
       "      <th>arbitration</th>\n",
       "      <th>lechin</th>\n",
       "      <th>otsuki</th>\n",
       "      <th>miti</th>\n",
       "      <th>iwc</th>\n",
       "      <th>witte</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>thai trade deficit widens in first quarter tha...</td>\n",
       "      <td>[corn, grain, rice, rubber, sugar, tin, trade]</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>indonesia sees cpo price rising sharply indone...</td>\n",
       "      <td>[palm-oil, veg-oil]</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 3335 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        article_text  \\\n",
       "0  thai trade deficit widens in first quarter tha...   \n",
       "1  indonesia sees cpo price rising sharply indone...   \n",
       "\n",
       "                                       categories  thai  trade  deficit  \\\n",
       "0  [corn, grain, rice, rubber, sugar, tin, trade]     1      2        2   \n",
       "1                             [palm-oil, veg-oil]     0      0        0   \n",
       "\n",
       "   first  quarter  thailand  billion  1987  ...    nogales  awf  amselco  amt  \\\n",
       "0      4        4         2        6     1  ...          0    0        0    0   \n",
       "1      0        0         0        0     0  ...          0    0        0    0   \n",
       "\n",
       "   arbitration  lechin  otsuki  miti  iwc  witte  \n",
       "0            0       0       0     0    0      0  \n",
       "1            0       0       0     0    0      0  \n",
       "\n",
       "[2 rows x 3335 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reminder of what we're working with.\n",
    "print('df shape:', df_bow.shape)\n",
    "df_bow.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How will we treat the outcome variable, 'categories'? We don't really want to look at all the extra (50+) categories we pulled in implicitly, so let's narrow it down to just the set of 15 original categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:11.710583Z",
     "start_time": "2018-03-02T17:51:11.705127Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['tin', 'coffee', 'gold', 'yen', 'rubber', 'palm-oil', 'lin-oil', 'rape-oil', 'wheat', 'soy-oil', 'nickel', 'cocoa', 'sugar', 'nzdlr', 'sorghum']\n"
     ]
    }
   ],
   "source": [
    "# This is our list of original categories.\n",
    "print(categories)\n",
    "categories_set = set(categories)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:14.291610Z",
     "start_time": "2018-03-02T17:51:14.265150Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['tin', 'rubber', 'sugar'], ['palm-oil'], ['palm-oil', 'coffee', 'rubber'], ['wheat'], ['gold']]\n"
     ]
    }
   ],
   "source": [
    "# Let's search through of 'categories' feature and see what matches up with those from the original group.\n",
    "categories_reduced = [list(categories_set.intersection(df_bow['categories'][index])) for index in df_bow.index]\n",
    "print(categories_reduced[:5])\n",
    "\n",
    "# Create a new feature in our DF.\n",
    "df_bow['categories_reduced'] = categories_reduced"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay, we've gotten rid of all the extra categories, but many articles still have multiple associations. It turns out Random Forest, Decision Tree, KNN, and MLP already support multilabel output, although metrics won't work for them. \n",
    "\n",
    "http://scikit-learn.org/stable/modules/multiclass.html\n",
    "\n",
    "Let's just run it on the reduced category set and see what happens. We do have to binarize our multilabel output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:16.580580Z",
     "start_time": "2018-03-02T17:51:16.532297Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:18.608528Z",
     "start_time": "2018-03-02T17:51:18.508410Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Array shapes: (721, 3333) (241, 3333) (721, 15) (241, 15)\n"
     ]
    }
   ],
   "source": [
    "mlb = MultiLabelBinarizer()\n",
    "\n",
    "# We have to exclude the category features we've been building.\n",
    "X = df_bow.loc[:, ~df_bow.columns.isin(['article_text', 'categories', 'categories_reduced'])]\n",
    "y = mlb.fit_transform(df_bow.categories_reduced)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=100)\n",
    "\n",
    "# We'll end up using some of these labeled outcomes later on for clusting.\n",
    "y_labels_train, y_labels_test = train_test_split(df_bow.categories_reduced, test_size=0.25, random_state=100)\n",
    "y_full_train, y_full_test = train_test_split(df_bow.categories, test_size=0.25, random_state=100)\n",
    "articles_train, articles_test = train_test_split(df_bow.article_text, test_size=0.25, random_state=100)\n",
    "\n",
    "print('Array shapes:', X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:28.392440Z",
     "start_time": "2018-03-02T17:51:28.233996Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Test set) (Model predictions)\n",
      "\n",
      "[[('wheat',) ('wheat',)]\n",
      " [('palm-oil',) ()]\n",
      " [('coffee',) ()]\n",
      " [('cocoa',) ()]\n",
      " [('wheat',) ('wheat',)]\n",
      " [('sugar',) ('sugar',)]\n",
      " [('sugar',) ()]\n",
      " [('cocoa',) ('cocoa',)]\n",
      " [('wheat',) ('wheat',)]\n",
      " [('sorghum',) ()]]\n"
     ]
    }
   ],
   "source": [
    "# Run Random Forest to get started.\n",
    "rfc = RandomForestClassifier().fit(X_train, y_train)\n",
    "y_pred = rfc.predict(X_test)\n",
    "\n",
    "# Print out results for the test and predicted sets.\n",
    "print('(Test set) (Model predictions)\\n')\n",
    "print(np.stack([mlb.inverse_transform(y_test), mlb.inverse_transform(y_pred)], axis=1)[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-26T18:37:10.926535Z",
     "start_time": "2018-02-26T18:37:10.919377Z"
    }
   },
   "source": [
    "It looks like we're doing well, but this isn't a very analytical way to rate our model. The biggest issue seems to be in not predicting any category in certain cases rather than mislabeling. We can use 'score', but keep in mind this will be 'subset accuracy', which means *every* label in the multilabel matrix must be predicted correctly for the example to count as correct. This is a harsh measure, so we don't expect our accuracy to be perfect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:40.262510Z",
     "start_time": "2018-03-02T17:51:40.218262Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.959778085992\n",
      "Test set score: 0.647302904564\n"
     ]
    }
   ],
   "source": [
    "print('Training set score:', rfc.score(X_train, y_train))\n",
    "print('Test set score:', rfc.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're overfitting the training set, but it's possible that 0.64 isn't so bad on the test set, since we're looking at subset accuracy. Let's see if we can do better with this or other models, and for now we'll keep using 'score' (subset accuracy) to measure effectiveness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:51:43.915724Z",
     "start_time": "2018-03-02T17:51:43.898572Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# These are the models that support multilabel output.\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multi Layer Perceptron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:56:41.832757Z",
     "start_time": "2018-03-02T17:52:08.811508Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/maxcalabro/anaconda3/lib/python3.6/site-packages/sklearn/neural_network/multilayer_perceptron.py:564: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.\n",
      "  % self.max_iter, ConvergenceWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best GridSearchCV params: {'hidden_layer_sizes': (200,)}\n",
      "Training set average cv score w/best parameters: 0.775312066574\n",
      "Test set score w/best parameters: 0.788381742739\n"
     ]
    }
   ],
   "source": [
    "mlp_params = {'hidden_layer_sizes':[(200,), (60, 60,), (30, 30, 30,)]}\n",
    "mlp_grid = GridSearchCV(MLPClassifier(), param_grid=mlp_params, cv=5).fit(X_train, y_train)\n",
    "print('Best GridSearchCV params:', mlp_grid.best_params_)\n",
    "print('Training set average cv score w/best parameters:', mlp_grid.best_score_)\n",
    "print('Test set score w/best parameters:', mlp_grid.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:57:01.320712Z",
     "start_time": "2018-03-02T17:56:41.836206Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best GridSearchCV params: {'max_depth': None, 'n_estimators': 100}\n",
      "Training set average cv score w/best parameters: 0.762829403606\n",
      "Test set score w/best parameters: 0.767634854772\n"
     ]
    }
   ],
   "source": [
    "rfc_params = {'max_depth':[10, 30, None],\n",
    "              'n_estimators':[10, 30, 100]}\n",
    "rfc_grid = GridSearchCV(RandomForestClassifier(), param_grid=rfc_params, cv=5).fit(X_train, y_train)\n",
    "print('Best GridSearchCV params:', rfc_grid.best_params_)\n",
    "print('Training set average cv score w/best parameters:', rfc_grid.best_score_)\n",
    "print('Test set score w/best parameters:', rfc_grid.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:57:06.331287Z",
     "start_time": "2018-03-02T17:57:01.323979Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best GridSearchCV params: {'max_depth': None, 'min_samples_split': 2}\n",
      "Training set average cv score w/best parameters: 0.912621359223\n",
      "Test set score w/best parameters: 0.904564315353\n"
     ]
    }
   ],
   "source": [
    "dtc_params = {'max_depth':[10, 30, None],\n",
    "              'min_samples_split':[2, 4, 8]}\n",
    "dtc_grid = GridSearchCV(DecisionTreeClassifier(), param_grid=dtc_params, cv=5).fit(X_train, y_train)\n",
    "print('Best GridSearchCV params:', dtc_grid.best_params_)\n",
    "print('Training set average cv score w/best parameters:', dtc_grid.best_score_)\n",
    "print('Test set score w/best parameters:', dtc_grid.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### K Nearest Neighbors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:57:58.107478Z",
     "start_time": "2018-03-02T17:57:06.334753Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best GridSearchCV params: {'n_neighbors': 4}\n",
      "Training set average cv score w/best parameters: 0.631067961165\n",
      "Test set score w/best parameters: 0.655601659751\n"
     ]
    }
   ],
   "source": [
    "knc_params = {'n_neighbors':[2, 4, 8, 16]}\n",
    "knc_grid = GridSearchCV(KNeighborsClassifier(), param_grid=knc_params, cv=5).fit(X_train, y_train)\n",
    "print('Best GridSearchCV params:', knc_grid.best_params_)\n",
    "print('Training set average cv score w/best parameters:', knc_grid.best_score_)\n",
    "print('Test set score w/best parameters:', knc_grid.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It looks like Decision Tree is doing the best -- even better than Random Forest. Interesting! 90% subset accuracy on the test set is quite remarkable. Let's look at that result a bit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Decision Tree Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T17:58:14.592326Z",
     "start_time": "2018-03-02T17:58:14.439916Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Test set) (Model predictions)\n",
      "\n",
      "[[('gold',) ('gold',)]\n",
      " [('rape-oil',) ('soy-oil',)]\n",
      " [('sugar',) ('sugar',)]\n",
      " [('wheat',) ('wheat',)]\n",
      " [('wheat',) ('wheat',)]\n",
      " [('wheat',) ('wheat',)]\n",
      " [('gold',) ('gold',)]\n",
      " [('sorghum',) ('sorghum',)]\n",
      " [('coffee',) ('coffee',)]\n",
      " [('sugar',) ('sugar',)]]\n"
     ]
    }
   ],
   "source": [
    "# Refit the best dtc.\n",
    "dtc = DecisionTreeClassifier(max_depth=30, min_samples_split=2).fit(X_train, y_train)\n",
    "y_pred = dtc.predict(X_test)\n",
    "\n",
    "# Print out results for the test and predicted sets.\n",
    "print('(Test set) (Model predictions)\\n')\n",
    "print(np.stack([mlb.inverse_transform(y_test), mlb.inverse_transform(y_pred)], axis=1)[10:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-01T17:09:34.904497Z",
     "start_time": "2018-03-01T17:09:32.947004Z"
    },
    "collapsed": true
   },
   "source": [
    "This is great -- not only is the accuracy impressive, the mistakes it's making are pretty minor (misclassifying one oil as another, including extra labels). This also gives insight into **why the Decision Tree is performing better than the Random Forest:** it isn't giving empty outputs the way RF did. Perhaps DT is less conservative in making predictions than RF because it doesn't have to find agreement over multiple estimators. \n",
    "\n",
    "Let's move on to unsupervised learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised BoW Model\n",
    "\n",
    "What if we try out some unsupervised models? The overlapping outcomes may cause some issues around the borders, but let's see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:01:25.969744Z",
     "start_time": "2018-03-02T18:01:25.871544Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans, MeanShift, AffinityPropagation, estimate_bandwidth\n",
    "from sklearn.decomposition import TruncatedSVD, PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:01:36.106726Z",
     "start_time": "2018-03-02T18:01:34.196731Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Run KMeans and grab the labels for training and test sets.\n",
    "n_clusters=10\n",
    "km = KMeans(n_clusters=n_clusters).fit(X_train)\n",
    "km_labels_train = km.labels_\n",
    "km_labels_test = km.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating an Evaluation Metric\n",
    "\n",
    "We're going to need a way to measure how well our clustering algorithm is doing, and there isn't an out-of-the-box solution for this one. I'm going to build an evaluator to take each cluster, determine which category is most common for the items in the cluster, and show how strongly that category dominates (simple percentages should be fine), and show the number of items in that cluster. **To test for cluster stability,** we will then predict outcomes on the test set, and compare those results to the cluster 'names' we determined with the training set. If the score is similar for the training and test set, we have good stability. If the training set is much higher, we have poor stability or overfitting. If the test set is higher, we got lucky with our label distributions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:04:52.011888Z",
     "start_time": "2018-03-02T18:04:51.997940Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function which outputs all the category labels for each cluster.\n",
    "# Takes input of the model_clusters, which is the result of model.labels_,\n",
    "# and true_labels, which is the categorical labels from y_train.\n",
    "#\n",
    "# Output is a dictionary with all the category labels for each cluster.\n",
    "\n",
    "def label_clusters(model_labels, true_labels):\n",
    "    # Create a dictionary.\n",
    "    clust_dict = {}\n",
    "    \n",
    "    # Set up each cluster as a key in the dictionary.\n",
    "    for label in set(model_labels):\n",
    "        clust_dict[label]=[]\n",
    "    \n",
    "    # Cycle through each example and append all the words to each key.\n",
    "    for i in range(len(model_labels)):\n",
    "        clust_dict[list(model_labels)[i]].append(list(true_labels)[i])\n",
    "    \n",
    "    # Make each key a list of strings rather than a list of lists.\n",
    "    for key in clust_dict.keys():\n",
    "        clust_dict[key]=[x[0] for x in clust_dict[key]]\n",
    "    \n",
    "    return clust_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:04:54.965790Z",
     "start_time": "2018-03-02T18:04:54.911620Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'tin', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee', 'coffee']\n"
     ]
    }
   ],
   "source": [
    "# Here's an example of how it works.\n",
    "clust_dict = label_clusters(km_labels_train, y_labels_train)\n",
    "print(clust_dict[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:01.107832Z",
     "start_time": "2018-03-02T18:05:01.104094Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function to find the most common label in a list.\n",
    "\n",
    "def find_common(lst):\n",
    "    return max(set(lst), key=lst.count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:03.456660Z",
     "start_time": "2018-03-02T18:05:03.448309Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'coffee'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# It works.\n",
    "find_common(clust_dict[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:10.279919Z",
     "start_time": "2018-03-02T18:05:10.272489Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function to return evaluation metrics from a list.\n",
    "\n",
    "def eval_list(lst, disp=True):\n",
    "    if disp:\n",
    "        print('Most common:', find_common(lst), \n",
    "              '\\nTotal most common:', lst.count(find_common(lst)), \n",
    "              '\\nTotal all items:', len(lst))\n",
    "    return [find_common(lst), \n",
    "            lst.count(find_common(lst)), \n",
    "            len(lst), \n",
    "            float(format(lst.count(find_common(lst))/len(lst), '.2f'))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:11.492338Z",
     "start_time": "2018-03-02T18:05:11.481083Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most common: coffee \n",
      "Total most common: 38 \n",
      "Total all items: 39\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['coffee', 38, 39, 0.97]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here's how it works.\n",
    "eval_list(clust_dict[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:35.862122Z",
     "start_time": "2018-03-02T18:05:35.814881Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Function to evaluate each cluster in a model.\n",
    "# Same inputs as label_clusters.\n",
    "# To run test set simultaneously (it uses the label from the training set), \n",
    "# include model_labels_test and true_labels_test.\n",
    "#\n",
    "# For large numbers of clusters, you may want show_clusters=False.\n",
    "#\n",
    "# Output is the percentage of points which are in a cluster where they are the most common element.\n",
    "\n",
    "def evaluate_clusters(model_labels, true_labels, model_labels_test=[], true_labels_test=[], show_clusters=True):\n",
    "    # Run the labeling function.\n",
    "    clust_dict = label_clusters(model_labels, true_labels)\n",
    "    \n",
    "    # Label the test set if provided.\n",
    "    if not true_labels_test is []:\n",
    "        clust_dict_test = label_clusters(model_labels_test, true_labels_test)\n",
    "    \n",
    "    # Keep track of how many points are well-labeled.\n",
    "    n_correct = 0\n",
    "    n_total = 0\n",
    "    n_correct_test = 0\n",
    "    n_total_test = 0\n",
    "    \n",
    "    # Tally for each key.\n",
    "    for key in clust_dict.keys():\n",
    "        \n",
    "        # Run our eval function\n",
    "        [label_common, n_common, n_all, common_pct] = eval_list(clust_dict[key], disp=False)\n",
    "        if show_clusters:\n",
    "            print([label_common, common_pct, n_all])\n",
    "        \n",
    "        # Add the correctly classified and total points to our tally.\n",
    "        n_correct += n_common\n",
    "        n_total += n_all\n",
    "        \n",
    "        # Add the correctly labeled test set results.\n",
    "        if not true_labels_test is []:\n",
    "            if not key in clust_dict_test.keys():\n",
    "                clust_dict_test[key] = []\n",
    "            n_correct_test += clust_dict_test[key].count(label_common)\n",
    "            n_total_test += len(clust_dict_test[key])\n",
    "   \n",
    "    return [n_correct/n_total, n_correct_test/n_total_test] if n_total_test else n_correct/n_total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:05:37.914772Z",
     "start_time": "2018-03-02T18:05:37.821807Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sugar', 1.0, 11]\n",
      "['coffee', 0.97, 39]\n",
      "['sorghum', 1.0, 1]\n",
      "['sorghum', 1.0, 1]\n",
      "['gold', 1.0, 62]\n",
      "['wheat', 0.2, 380]\n",
      "['wheat', 0.94, 35]\n",
      "['wheat', 0.5, 152]\n",
      "['cocoa', 0.67, 15]\n",
      "['wheat', 0.44, 25]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.44105409153952846, 0.4605809128630705]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_clusters(km_labels_train, y_labels_train, km_labels_test, y_labels_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:39:55.715312Z",
     "start_time": "2018-02-28T22:39:50.707472Z"
    },
    "collapsed": true
   },
   "source": [
    "#### What is our metric actually doing?\n",
    "\n",
    "Great! We now have an intuitive method for evaluating our clusters. Specifically, evaluate_clusters( ) takes the output from our model, compares it to the category labels from our data set, and returns the percentage of examples which are in a cluster where they are the most common element. So with a single cluster of three elements, if 2 elements are the same, the score would be 0.67. A more complicated example would be:\n",
    "\n",
    "    Cluster 0: [X, X, Y, Y, Y]  <--- This cluster will be labeled 'Y'\n",
    "    Cluster 1: [Z, Y, X, X]     <--- This cluster will be labeled 'X'\n",
    "    \n",
    "This example would have a score of \n",
    "\n",
    "    (3+2)/(5+4) = .56\n",
    "    \n",
    "Or in words,\n",
    "    \n",
    "    (# correct from cluster 0 + # correct from cluster 1)/(total # cluster 0 + total # cluster 1)\n",
    "    \n",
    "Note that for multilabeled data, it is impossible to achieve a score of 1.0 for the whole data set because elements will introduce labels which are not the most common into their cluster (for instance, a ['wheat', 'sugar'] element goes into a 'wheat' cluster, and the 'sugar' label lowers the score). \n",
    "\n",
    "This is also not a perfect evaluation metric because articles with more labels will be weighted more heavily in the analysis. That's fine -- we're only comparing this metric to itself within this one data set. If we transferred it to a different data set, the values we consider to be 'good' maybe be substantially different (specifically, the more tags the examples have on average, the lower the score we expect).\n",
    "\n",
    "The \"test set\" output from the function is the same metric, but using the common label from the training set rather than from the test set. This tests cluster stability.\n",
    "\n",
    "Let's see how some models do."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running Unsupervised BoW Models\n",
    "\n",
    "Let's start with some different numbers of clusters for KMeans."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### KMeans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:12:22.761018Z",
     "start_time": "2018-03-02T18:11:56.553849Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 Clusters: [0.27323162274618584, 0.2697095435684647]\n",
      "4 Clusters: [0.3092926490984743, 0.3029045643153527]\n",
      "8 Clusters: [0.3342579750346741, 0.34439834024896265]\n",
      "16 Clusters: [0.44105409153952846, 0.44398340248962653]\n",
      "32 Clusters: [0.5561719833564494, 0.5560165975103735]\n",
      "64 Clusters: [0.6685159500693482, 0.6058091286307054]\n",
      "128 Clusters: [0.7115117891816921, 0.6390041493775933]\n"
     ]
    }
   ],
   "source": [
    "for n in [2, 4, 8, 16, 32, 64, 128]:\n",
    "    km = KMeans(n_clusters=n).fit(X_train)\n",
    "    km_labels_train = km.labels_\n",
    "    km_labels_test = km.predict(X_test)\n",
    "    print('{} Clusters:'.format(n), evaluate_clusters(km_labels_train, \n",
    "                                                      y_labels_train, \n",
    "                                                      km_labels_test, \n",
    "                                                      y_labels_test,\n",
    "                                                      show_clusters=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:38:52.528550Z",
     "start_time": "2018-02-28T22:38:52.451117Z"
    },
    "collapsed": true
   },
   "source": [
    "So, we do better and better as we increase the number of clusters, but we don't want to overfit the data by giving every sentence its own cluster. We have 15 different categories which are possible outputs, so is it fair to stick with 32 clusters. With more clusters, we also lose stability, as the test set score drops. Let's see how our clusters are defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:13:37.443696Z",
     "start_time": "2018-03-02T18:13:33.989531Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['coffee', 1.0, 49]\n",
      "['sugar', 0.28, 334]\n",
      "['sorghum', 1.0, 1]\n",
      "['sugar', 0.55, 11]\n",
      "['wheat', 1.0, 1]\n",
      "['sorghum', 1.0, 2]\n",
      "['sugar', 1.0, 1]\n",
      "['sorghum', 1.0, 1]\n",
      "['cocoa', 1.0, 8]\n",
      "['yen', 1.0, 1]\n",
      "['coffee', 1.0, 1]\n",
      "['palm-oil', 1.0, 1]\n",
      "['wheat', 0.93, 129]\n",
      "['wheat', 0.67, 21]\n",
      "['wheat', 0.61, 41]\n",
      "['rubber', 1.0, 4]\n",
      "['coffee', 1.0, 4]\n",
      "['yen', 1.0, 9]\n",
      "['tin', 1.0, 1]\n",
      "['wheat', 1.0, 1]\n",
      "['sorghum', 1.0, 3]\n",
      "['sugar', 1.0, 1]\n",
      "['sugar', 1.0, 1]\n",
      "['nzdlr', 1.0, 1]\n",
      "['sugar', 1.0, 1]\n",
      "['coffee', 1.0, 5]\n",
      "['gold', 1.0, 64]\n",
      "['sugar', 1.0, 1]\n",
      "['wheat', 1.0, 5]\n",
      "['rubber', 1.0, 1]\n",
      "['wheat', 0.67, 15]\n",
      "['cocoa', 1.0, 2]\n",
      "32 Clusters: 0.6061026352288488\n"
     ]
    }
   ],
   "source": [
    "n_clusters=32\n",
    "km = KMeans(n_clusters=n_clusters).fit(X_train)\n",
    "km_labels_train = km.labels_\n",
    "print('{} Clusters:'.format(n_clusters), evaluate_clusters(km_labels_train, y_labels_train, show_clusters=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:39:46.562235Z",
     "start_time": "2018-02-28T22:39:20.786061Z"
    },
    "collapsed": true
   },
   "source": [
    "Like our initial example of 10 clusters, we continue to have one large cluster with very poor separation ('wheat': 0.28, 334), and a lot of clusters that are only one example. A score of 0.61 isn't terrible, though. Let's try some other models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### MeanShift"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:18:10.685278Z",
     "start_time": "2018-03-02T18:16:55.327096Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "109 Clusters: [0.40221914008321774, 0.3236514522821577]\n"
     ]
    }
   ],
   "source": [
    "bandwidth = estimate_bandwidth(X_train, quantile=0.5, n_samples=500)\n",
    "\n",
    "ms = MeanShift(bandwidth=bandwidth).fit(X_train)\n",
    "ms_labels_train = ms.labels_\n",
    "ms_labels_test = ms.predict(X_test)\n",
    "n_clusters = ms.cluster_centers_.shape[0]\n",
    "\n",
    "print('{} Clusters:'.format(n_clusters), evaluate_clusters(ms_labels_train, \n",
    "                                                           y_labels_train, \n",
    "                                                           ms_labels_test, \n",
    "                                                           y_labels_test, \n",
    "                                                           show_clusters=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:44:16.566225Z",
     "start_time": "2018-02-28T22:41:35.009331Z"
    },
    "collapsed": true
   },
   "source": [
    "Yuck -- we're getting a low score and a lot of clusters. Digging into the this result, we're creating a lot of clusters with a single point, and leaving one large cluster that is very poorly differentiated. Even so it does really badly -- evaluation of 0.40 on training and 0.33 on test. Changing the bandwidth had no impact on this result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:44:16.566225Z",
     "start_time": "2018-02-28T22:41:35.009331Z"
    },
    "collapsed": true
   },
   "source": [
    "#### Affinity Propagation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:19:22.522789Z",
     "start_time": "2018-03-02T18:19:20.289668Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "151 Clusters: [0.6712898751733704, 0.5684647302904564]\n"
     ]
    }
   ],
   "source": [
    "ap = AffinityPropagation(damping=0.5).fit(X_train)\n",
    "ap_labels_train = ap.labels_\n",
    "ap_labels_test = ap.predict(X_test)\n",
    "n_clusters = ap.cluster_centers_.shape[0]\n",
    "\n",
    "print('{} Clusters:'.format(n_clusters), evaluate_clusters(ap_labels_train, \n",
    "                                                           y_labels_train, \n",
    "                                                           ap_labels_test, \n",
    "                                                           y_labels_test, \n",
    "                                                           show_clusters=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:44:16.566225Z",
     "start_time": "2018-02-28T22:41:35.009331Z"
    },
    "collapsed": true
   },
   "source": [
    "This score (0.68 training, 0.58 test) is pretty good, but we're creating way too many clusters (151) and overfitting, as evidenced by the lower test score. For reference, when we allowed 128 clusters for KMeans, our training and test values were 0.71 and 0.64, respectively.\n",
    "\n",
    "Overall, KMeans is doing the best. We're able to control the number of clusters to prevent overfitting, and we're getting the highest evaluation score with fewer clusters than the other algorithms. I do think somewhere around 30-50 clusters is the sweet spot of maximizing accuracy and cluster stability without overfitting too much.\n",
    "\n",
    "Moving on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSA with BoW\n",
    "\n",
    "How about Latent Semantic Analysis with our Bag of Words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:34:40.954855Z",
     "start_time": "2018-03-02T18:34:40.928306Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.preprocessing import Normalizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:34:44.513063Z",
     "start_time": "2018-03-02T18:34:43.887574Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance explained with 30 components: 0.634552666295\n"
     ]
    }
   ],
   "source": [
    "# Build and fit our decomposition model. \n",
    "# Let's use 30 components to maintain consistency with our other models.\n",
    "n_components = 30\n",
    "svd = TruncatedSVD(n_components)\n",
    "lsa = make_pipeline(svd, Normalizer(copy=False))\n",
    "\n",
    "X_train_lsa = lsa.fit_transform(X_train)\n",
    "\n",
    "print('Variance explained with {} components:'.format(n_components), svd.explained_variance_ratio_.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:34:57.411017Z",
     "start_time": "2018-03-02T18:34:57.379664Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Component 0:\n",
      "categories_reduced\n",
      "[sorghum, soy-oil]    0.998766\n",
      "[coffee]              0.986185\n",
      "[coffee]              0.984603\n",
      "[coffee]              0.971402\n",
      "[sugar]               0.968813\n",
      "Name: 0, dtype: float64\n",
      "Component 1:\n",
      "categories_reduced\n",
      "[wheat]    0.825658\n",
      "[sugar]    0.821258\n",
      "[wheat]    0.807990\n",
      "[sugar]    0.782224\n",
      "[sugar]    0.776996\n",
      "Name: 1, dtype: float64\n",
      "Component 2:\n",
      "categories_reduced\n",
      "[wheat]                      0.460561\n",
      "[sorghum, wheat, soy-oil]    0.432984\n",
      "[wheat]                      0.428114\n",
      "[wheat]                      0.422438\n",
      "[wheat]                      0.421776\n",
      "Name: 2, dtype: float64\n",
      "Component 3:\n",
      "categories_reduced\n",
      "[wheat]                      0.692269\n",
      "[sorghum, wheat, soy-oil]    0.684675\n",
      "[wheat, soy-oil]             0.475268\n",
      "[soy-oil]                    0.330399\n",
      "[gold]                       0.329753\n",
      "Name: 3, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Make a Dataframe with categories as indices and components as features\n",
    "categories_by_component=pd.DataFrame(X_train_lsa,index=y_labels_train)\n",
    "categories_by_component.shape\n",
    "\n",
    "# Show the highest values for a few components, along with their articles\n",
    "for i in range(4):\n",
    "    print('Component {}:'.format(i))\n",
    "    print(categories_by_component.loc[:,i].sort_values(ascending=False)[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:45:09.687563Z",
     "start_time": "2018-02-28T22:45:09.674514Z"
    },
    "collapsed": true
   },
   "source": [
    "This is great! Many of these components are excellent. I'm curious about components 0 and 1 -- they have very high scores, but different categories. Maybe there is some other similarity in the text? Let's look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T18:44:49.334691Z",
     "start_time": "2018-03-02T18:44:49.310020Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Component 0:\n",
      "- net change in export commitments -- usda the u.s. agriculture department gave the net change in export commitments, including sales, cancellations, foreign purchases and cumulative exports, in the cur \n",
      "- brazilian coffee rainfall the following rainfall was recorded in the areas over the past 24 hours parana state: umuarama nil, paranavai nil, londrina nil, maringa nil. sao paulo state presidente prude \n",
      "- brazilian coffee rainfall the following rainfall was recorded in the areas over the past 24 hours parana state: umuarama nil, paranavai nil, londrina nil, maringa nil. sao paulo state presidente prude \n",
      "- brazilian coffee rainfall the following rainfall was recorded in the areas over past 72 hours parana state: umuarama nil, paranavai 1.5 millimetres, londrina nil, maringa nil. sao paulo state: preside \n",
      "- sugar quota imports detailed -- usda the u.s. agriculture department said cumulative sugar imports from individual countries during the 1987 quota year, which began january 1, 1987 and ends december 3 \n",
      "\n",
      "Component 1:\n",
      "- sichuan braced to fight drought the sichuan government has ordered that any work or meeting which interferes with the fight against drought must be cancelled or postponed to save time, energy and manp \n",
      "- thai sugar production continues high in feb thai sugar production continued at a high level in february, latest figures received by the international sugar organization (iso) show. the figures show st \n",
      "- pakistan not seen as major wheat exporter pakistan is not emerging as a major wheat exporter as world market prospects are not good enough, sartaj aziz, special assistant on food and agriculture to th \n",
      "- china seen unlikely to raise sugar imports china will not increase sugar imports substantially this year because of foreign exchange constraints and large stocks, despite falling production and rising \n",
      "- thai sugar production increases thai sugar production totalled 960,788 tonnes in january, an increase of 12.7 pct on january 1986, according to figures received by the international sugar organization \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "articles_by_component=pd.DataFrame(X_train_lsa,index=articles_train)\n",
    "\n",
    "print('Component 0:')\n",
    "[print('-', index[:200], '') for index in articles_by_component.loc[:,0].sort_values(ascending=False).index[0:5]]\n",
    "\n",
    "print('\\nComponent 1:')\n",
    "[print('-', index[:200], '') for index in articles_by_component.loc[:,1].sort_values(ascending=False).index[0:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T22:45:37.777090Z",
     "start_time": "2018-02-28T22:45:37.740598Z"
    },
    "collapsed": true
   },
   "source": [
    "Ah -- so Component 0 is full of production reports that contain the word 'nil' numerous times. Component 1 seems to talk about production and tonnes quite a bit (I've shortened the articles for display). It is understandable that there would be a lot of crossover in terms for some of these similar commodity-related categories. \n",
    "\n",
    "This may actually be our most effective unsupervised method thus far. Let's see how we do with tfidf!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tfidf\n",
    "\n",
    "Let's see if we can do better than our Bag of Words by using tfidf. It's likely this will get rid of some of the noise we were seeing in our LSA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:10:30.399358Z",
     "start_time": "2018-03-02T19:10:30.395206Z"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:10:32.011473Z",
     "start_time": "2018-03-02T19:10:31.209211Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Articles and Features: (962, 2003)\n"
     ]
    }
   ],
   "source": [
    "# Create the model, fit, and transform the original articles. Each article is one row.\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=0.4,\n",
    "                             min_df=10,\n",
    "                             stop_words='english',\n",
    "                             lowercase=True,\n",
    "                             ngram_range=(1, 2),\n",
    "                             use_idf=True,\n",
    "                             norm=u'l2',\n",
    "                             smooth_idf=True)\n",
    "\n",
    "articles_tfidf = vectorizer.fit_transform(articles)\n",
    "print('Number of Articles and Features:', articles_tfidf.get_shape())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've tuned the model to run well by adjusting max_df, min_df, and ngram_range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:10:36.143557Z",
     "start_time": "2018-03-02T19:10:34.217550Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original Article:\n",
      " cocoa council meeting ends after agreeing rules the international cocoa organization (icco) council adjourned after agreeing buffer stock rules for the 1986 international cocoa agreement, an icco spokesman said. the buffer stock will begin operations immediately, he said. he confirmed delegate repor ...\n",
      "\n",
      "\n",
      "Tfidf Vector:\n",
      " {'non members': 0.12324648290357955, 'intermediate': 0.12510600952205911, 'icco council': 0.11568637363883798, 'ends': 0.12324648290357955, 'council meeting': 0.1132386717919496, 'different': 0.10542185626231504, 'differentials': 0.11210519428187977, 'cocoa agreement': 0.11842267150249437, 'nearby': 0.10900303561927323, 'organization icco': 0.10231969759970849, 'cocoa organization': 0.10091128574959078, 'organization': 0.07778581918204501, 'icco': 0.19091113475812635, 'operations': 0.21084371252463008, 'reports': 0.099583399736052097, 'day': 0.084469157201609746, 'offer': 0.083480425833266203, 'fixed': 0.10626673775561681, 'limited': 0.10231969759970849, 'immediately': 0.10626673775561681, 'positions': 0.10900303561927323, 'stock rules': 0.10460662592886884, 'international cocoa': 0.19312409400090944, 'stock manager': 0.10381903590872847, 'buffer stock': 0.25442493554041468, 'means': 0.10714350900079367, 'delegate': 0.10805467208140483, 'selling': 0.092421041022135719, 'rules': 0.18984296446935484, 'manager': 0.091040535098007788, 'buffer': 0.25442493554041468, 'cocoa': 0.40960333497249041, 'pct total': 0.11443030122372971, '15 pct': 0.10305727321819995, '15': 0.06904951728493898, 'begin': 0.10381903590872847, 'non': 0.088503652026082943, 'maximum': 0.096562047000454718, 'stock': 0.23112106112283953, 'buying': 0.081324950236857943, 'spokesman said': 0.089318882359529173, 'meeting': 0.072804717534936808, 'spokesman': 0.078296426122721419, 'council': 0.16148825593921959, 'members': 0.085856535387463018, 'agreement': 0.065969040080362598, 'confirmed': 0.1110244561374633, '40 pct': 0.11842267150249437, '40': 0.081920666994498084, 'total': 0.067216678412752279, 'forward': 0.1110244561374633, 'according': 0.083480425833266203, '1986': 0.054936463311661798, 'international': 0.11744632830068202, 'purchases': 0.08515249118772486, 'pct': 0.1067681288449082, 'trade': 0.057743697659724688}\n",
      "\n",
      "\n",
      "Original Categories:\n",
      " ['cocoa']\n"
     ]
    }
   ],
   "source": [
    "# Split Tfidf results into train and test.\n",
    "X_tfidf_train, X_tfidf_test = train_test_split(articles_tfidf, test_size=0.25, random_state=100)\n",
    "\n",
    "# Make a version we can read.\n",
    "X_tfidf_train_csr = X_tfidf_train.tocsr()\n",
    "\n",
    "# Number of training articles\n",
    "n = X_tfidf_train.get_shape()[0]\n",
    "\n",
    "# A list of dictionaries, one per article\n",
    "tfidf_byarticle = [{} for _ in range(0,n)]\n",
    "\n",
    "# List of features\n",
    "terms = vectorizer.get_feature_names()\n",
    "\n",
    "# For each article, lists the feature words and their tf-idf scores\n",
    "for i, j in zip(*X_tfidf_train_csr.nonzero()):\n",
    "    tfidf_byarticle[i][terms[j]] = X_tfidf_train_csr[i, j]\n",
    "    \n",
    "print('Original Article:\\n', list(articles_train)[5][:300], '...')\n",
    "print('\\n\\nTfidf Vector:\\n', list(tfidf_byarticle)[5])\n",
    "print('\\n\\nOriginal Categories:\\n', list(y_labels_train)[5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LSA with Tfidf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:11:57.610552Z",
     "start_time": "2018-03-02T19:11:57.560275Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance explained with 30 components: 0.30173615291\n"
     ]
    }
   ],
   "source": [
    "# Build and fit our decomposition model. \n",
    "# We can alter the number of components until we're happy with our explained variance.\n",
    "n_components = 30\n",
    "svd = TruncatedSVD(n_components)\n",
    "lsa = make_pipeline(svd, Normalizer(copy=False))\n",
    "\n",
    "X_train_lsa = lsa.fit_transform(X_tfidf_train)\n",
    "\n",
    "print('Variance explained with {} components:'.format(n_components), svd.explained_variance_ratio_.sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:13:34.905653Z",
     "start_time": "2018-03-02T19:13:34.882639Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(721, 30)\n",
      "\n",
      "Component 0:\n",
      "- iwc says effect of lower support prices limited efforts by governments to control wheat surpluses by cutting support prices have met with only partial success, the international wheat council (iwc) sa \n",
      "- ec driving to capture brazil wheat market - usda the european community, ec, sold 75,000 tonnes of soft wheat at a subsidized price of between 85 and 89 dlrs per tonne fob for march delivery in a cont \n",
      "- pakistan not seen as major wheat exporter pakistan is not emerging as a major wheat exporter as world market prospects are not good enough, sartaj aziz, special assistant on food and agriculture to th \n",
      "- nickel prices unlikely to rise much - shearson nickel prices are unlikely to rise significantly from current levels unless further steps are taken to reduce production, shearson lehman brothers said i \n",
      "- australian government to pay subsidies--usda the australian government will likely reimburse the australian wheat board, awb, about 132 mln (u.s.) dlrs to pay wheat farmers for their 1986/87 crop, the \n",
      "\n",
      "Component 1:\n",
      "- coffee quota talks continue, no accord seen likely the international coffee organization (ico ) council talks on reintroducing export quotas continued with an extended session lasting late into sunday \n",
      "- u.s. sees no new coffee agreement talks soon the united states does not expect the executive board meeting of the international coffee organization, ico, to call for a new round of negotiations on rei \n",
      "- coffee traders expect selloff after ico talks fail the failure of the international coffee organization (ico) to reach agreement on coffee export quotas could trigger a massive selloff in london coffe \n",
      "- ico talks on coffee quotas to resume at noon talks on coffee export quotas at the international coffee organization (ico) special council session will resume at noon gmt today, following a last minute \n",
      "- coffee quota talks continue but no agreement yet coffee quota talks at the international coffee organization council meeting here continued this afternoon, but producers and consumers still had not re \n"
     ]
    }
   ],
   "source": [
    "# Make a Dataframe with articles as indices and components as features\n",
    "articles_by_component=pd.DataFrame(X_train_lsa,index=articles_train)\n",
    "print(articles_by_component.shape)\n",
    "\n",
    "# Show the highest values for a few components, along with their articles\n",
    "for i in range(2):\n",
    "    print('\\nComponent {}:'.format(i))\n",
    "    [print('-', index[:200], '') for index in articles_by_component.loc[:,i].sort_values(ascending=False).index[0:5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's cool, but it would be easier to see how we're doing if we looked at the 'Category' labels like we did with BoW."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:14:24.165648Z",
     "start_time": "2018-03-02T19:14:24.141671Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Component 0:\n",
      "categories_reduced\n",
      "[wheat]     0.747260\n",
      "[wheat]     0.730438\n",
      "[wheat]     0.712247\n",
      "[nickel]    0.696459\n",
      "[wheat]     0.681663\n",
      "Name: 0, dtype: float64\n",
      "Component 1:\n",
      "categories_reduced\n",
      "[coffee]    0.676703\n",
      "[coffee]    0.675933\n",
      "[coffee]    0.669917\n",
      "[coffee]    0.668982\n",
      "[coffee]    0.668293\n",
      "Name: 1, dtype: float64\n",
      "Component 2:\n",
      "categories_reduced\n",
      "[gold]    0.820549\n",
      "[gold]    0.789516\n",
      "[gold]    0.787472\n",
      "[gold]    0.787414\n",
      "[gold]    0.784667\n",
      "Name: 2, dtype: float64\n",
      "Component 3:\n",
      "categories_reduced\n",
      "[sugar]    0.579892\n",
      "[sugar]    0.561904\n",
      "[sugar]    0.552595\n",
      "[sugar]    0.546670\n",
      "[sugar]    0.545400\n",
      "Name: 3, dtype: float64\n",
      "Component 4:\n",
      "categories_reduced\n",
      "[cocoa]    0.574239\n",
      "[cocoa]    0.574069\n",
      "[cocoa]    0.562960\n",
      "[cocoa]    0.556698\n",
      "[cocoa]    0.554630\n",
      "Name: 4, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# Make a Dataframe with categories as indices and components as features\n",
    "categories_by_component=pd.DataFrame(X_train_lsa,index=y_labels_train)\n",
    "categories_by_component.shape\n",
    "\n",
    "# Show the highest values for a few components, along with their articles\n",
    "for i in range(5):\n",
    "    print('Component {}:'.format(i))\n",
    "    print(categories_by_component.loc[:,i].sort_values(ascending=False)[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! After some tweaking of tfidf parameters, we're doing fantastic with our component decomposition. It's hard to tell if it's better or worse than with BoW, so let's run an unsupervised model and use our evaluation metric on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KMeans\n",
    "\n",
    "KMeans did the best with our BoW analysis, let's try it on tfidf. This will continue to allow us to choose the number of clusters and make sure we're not overfitting or creating instability in our clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:16:31.475039Z",
     "start_time": "2018-03-02T19:16:10.143609Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 Clusters: [0.3925104022191401, 0.38589211618257263]\n",
      "4 Clusters: [0.6144244105409153, 0.5975103734439834]\n",
      "8 Clusters: [0.7226074895977809, 0.7261410788381742]\n",
      "16 Clusters: [0.7600554785020804, 0.7759336099585062]\n",
      "32 Clusters: [0.8238557558945908, 0.8008298755186722]\n",
      "64 Clusters: [0.8460471567267683, 0.8547717842323651]\n",
      "128 Clusters: [0.8765603328710125, 0.8298755186721992]\n"
     ]
    }
   ],
   "source": [
    "for n in [2, 4, 8, 16, 32, 64, 128]:\n",
    "    km = KMeans(n_clusters=n).fit(X_tfidf_train)\n",
    "    km_labels_train = km.labels_\n",
    "    km_labels_test = km.predict(X_tfidf_test)\n",
    "    print('{} Clusters:'.format(n), evaluate_clusters(km_labels_train, \n",
    "                                                      y_labels_train, \n",
    "                                                      km_labels_test, \n",
    "                                                      y_labels_test,\n",
    "                                                      show_clusters=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Wow! We're doing great -- much better than with BoW! With 32 clusters, we're up from ~55% to over 80% accuracy with our evaluation metric. We could maybe push the number of clusters up slightly without overfitting too much while maintaining cluster stability, but we don't do much better with 64 clusters than with 32. Let's see how our clusters are actually defined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:18:53.172698Z",
     "start_time": "2018-03-02T19:18:50.387393Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['wheat', 0.71, 66]\n",
      "['coffee', 1.0, 18]\n",
      "['cocoa', 0.97, 33]\n",
      "['gold', 0.8, 10]\n",
      "['yen', 0.91, 11]\n",
      "['wheat', 1.0, 23]\n",
      "['coffee', 0.44, 9]\n",
      "['sugar', 0.95, 39]\n",
      "['gold', 0.62, 13]\n",
      "['wheat', 0.63, 30]\n",
      "['sugar', 0.96, 26]\n",
      "['palm-oil', 0.45, 11]\n",
      "['coffee', 1.0, 31]\n",
      "['sugar', 0.47, 34]\n",
      "['rubber', 0.97, 32]\n",
      "['wheat', 0.68, 19]\n",
      "['yen', 0.95, 37]\n",
      "['wheat', 1.0, 15]\n",
      "['gold', 1.0, 28]\n",
      "['coffee', 0.3, 10]\n",
      "['palm-oil', 1.0, 20]\n",
      "['wheat', 1.0, 16]\n",
      "['coffee', 1.0, 35]\n",
      "['gold', 0.82, 28]\n",
      "['sugar', 1.0, 17]\n",
      "['wheat', 1.0, 15]\n",
      "['sorghum', 0.42, 12]\n",
      "['wheat', 0.88, 8]\n",
      "['wheat', 1.0, 10]\n",
      "['wheat', 0.47, 15]\n",
      "['tin', 0.94, 16]\n",
      "['gold', 0.68, 34]\n",
      "32 Clusters: 0.8335644937586685\n"
     ]
    }
   ],
   "source": [
    "n_clusters=32\n",
    "km = KMeans(n_clusters=n_clusters).fit(X_tfidf_train)\n",
    "km_labels_train = km.labels_\n",
    "print('{} Clusters:'.format(n_clusters), evaluate_clusters(km_labels_train, y_labels_train, show_clusters=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the clusters we've created, we're doing substantially better than we did with BoW. Most clusters are about the same size (15-30 elements), most have quite high frequency of the most common category (0.80+), and only a couple have low accuracy. Remember also that because we're looking at multilabel output, it's impossible for us to actually get to 1.0 accuracy over the whole data set -- there will always be some tags that have come along for the ride. For example, an article tagged as 'wheat', 'sugar', and 'coffee' will bring all three of those tags into the cluster, making it impossible to label the cluster with 100% accuracy.\n",
    "\n",
    "### KMeans or LSA?\n",
    "\n",
    "Which is better, KMeans or LSA? Well, it depends on what our needs and our goals are. KMeans should run faster and scale better to larger data sets. We're only looking at 1,000 examples and 2,000 features here. In production, we might have millions of examples and 10x more features. Because we retain feature names (while LSA obscures them), this may allow us to automatically create new tags based on the features that are most important for each cluster. Also, we *could* extract a score for how well each example fits into each cluster to allow multilabel classification, but otherwise we're relatively stuck having each example tied to a single cluster.\n",
    "\n",
    "LSA is more flexible. Every example gets a score for each component, so we could set a threshold and apply a label for every component where an example meets that threshold. It's a bit easier to peer inside and see how strongly examples are exhibit the properties of each component, and to ignore components that may be clustering by some characteristic that we're actually not interested in. Determining what that characteristic is, however, may be a bit tricky, especially with unlabeled data.\n",
    "\n",
    "\n",
    "### Why is tfidf doing so much better?\n",
    "\n",
    "Remember how our unsupervised BoW models kept creating one large cluster that was poorly differentiated? I suspect that cluster was sticking together for a reason: high correlation among common, unimportant words. When we looked at BoW LSA, we saw that one of the best groups was one that contained the word 'nil' tens of times. The whole point of tfidf (literally the *idf* part: inverse document frequency) is to discount words that are common throughout the entire corpus. By discounting common words like 'nil', 'tonnes', 'millions', etc, we're able to differentiate clusters by the words that are actually important to the individual articles. Basically: tfidf is doing exactly what it does best, and this is a clear example of why we use it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity\n",
    "\n",
    "Finally, let's take a look at some articles which are considered similar to each other are see if we can figure out why."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:44:51.291448Z",
     "start_time": "2018-03-02T19:44:50.903720Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAAD3CAYAAACgsbc4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGpdJREFUeJzt3X9UVGX+B/D3/GAAGcRVikoBkdOYR1tRO2YZpSZZmOsR\n9DtgQau2mdbR9Ut7BH8gKvFj3VNtbv48u+uKleSPLfFbeiSwktZScgz2KKWSJXWwVVwdCIeZ+3z/\n8Di7qAyDc+/lXnu/PPccZy5zP8+kfvo8z3Pv8xiEEAJERBQQY3c3gIjoVsBkSkQkAyZTIiIZMJkS\nEcmAyZSISAZMpkREMmAyJaKftaNHjyIjI+O69ysqKpCamgq73Y533nmn0+uYlWgcEZEebNy4Ebt2\n7UJoaGi799va2lBYWIjt27cjNDQU6enpGDduHCIjIzu8FitTIvrZiomJwerVq697/+TJk4iJiUFE\nRAQsFgtGjBiBQ4cO+byW7JVp279OyX3JG7rnnqmqxAGAmJA+qsU653aqEqfxpyZV4gDAHaG9VYs1\nODhKtVjRhhBV4lyCR5U4ALD74jHVYgHAt+drAvp8V/JNUOSA696bMGECzpw5c937TqcT4eHh3tdh\nYWFwOn3/22Q3n4j0S1LmfzRWqxXNzc3e183Nze2S642wm09E+iUk/48uiI+Px+nTp3HhwgW4XC4c\nPnwYw4YN8/kZVqZEpF9S15JkZ8rKytDS0gK73Y7s7GzMmjULQgikpqYiKsr3EJJB7lWjOGYaGI6Z\nBoZjpoHR25ip6/t/+v2zlrsGBxSrM6xMiUi/PO7uboEXkykR6ZdCE1A3g8mUiPSrixNLSmIyJSL9\nknkCKhB+J1NJkmA08k4qItIOoZfK9LvvvkNhYSFqa2thNpshSRJsNhtycnIQFxenVhuJiG5ML5Xp\n4sWLkZWVhaFDh3rfczgcyMnJwdatWxVvHBGRT5627m6Bl89k6nK52iVSAEhISFC0QUREftNLN3/g\nwIHIyclBYmIiwsPD0dzcjI8++ggDBw5Uq31ERB3TSzc/Ly8P5eXlqK6uhtPphNVqxdixY5GUlKRW\n+4iIOqaXytRgMCApKYnJk4i0SS+VKRGRlglJJxNQRESaxsqUiEgGehkzJSLSNC50QkQkg1u5MlVr\n0ebjx7erEgcA4m2TVYt1rvWSKnEGRvRTJQ4ABBuDVIu1YZp6/7hSSy+oEufby+dUiQMAdTsXqBZL\nFhwzJSKSAReHJiKSAStTIqLACcEJKCKiwLEyJSKSwa08m09EpBpWpkREMuBsPhGRDNjNJyKSAbv5\nREQyYDIlIpKBXrr5GRkZaGtrv/iqEAIGg4G7kxJR99PLBNRLL72EJUuW4I033oDJZFKrTURE/tFL\nN3/o0KGYPHky6urquA8UEWmPXrr5APDss8+q0Q4ioq7TS2VKRKRpGkqmxu5uABHRTRPC/+MakiQh\nNzcXdrsdGRkZOH36dLvzu3btwpQpU5Camoq33nqr06awMiUi/XLf/Gx+eXk5XC4XSktL4XA4UFRU\nhLVr13rP//73v8fu3bvRo0cPTJw4ERMnTkRERESH12MyJSL9CmACqrq6GomJiQCAhIQE1NbWtjs/\ncOBAXLp0CWaz2XtLqC9MpkSkXwGMmTqdTlitVu9rk8kEt9sNs/lKWrz77ruRmpqK0NBQJCUloWfP\nnj6vxzFTItKvAMZMrVYrmpubva8lSfIm0uPHj2P//v348MMPUVFRgfPnz+ODDz7w2RTZK9OYkD5y\nX/KG1Nwx9ORX76kW6+kR/6tKnEaPU5U4ANDouqharKGb/q1aLEmlLTPKIqJViQMAkzJKVYsFAOXf\nzQzsAgFUpsOHD0dlZSWSk5PhcDhgs9m858LDwxESEoLg4GCYTCb07t0bFy/6/nvMbj4R6VcAyTQp\nKQlVVVVIS0uDEAIFBQUoKytDS0sL7HY77HY7pk+fjqCgIMTExGDKlCk+r8dkSkS6JTw33zswGo1Y\nsWJFu/fi4+O9v09PT0d6errf12MyJSL90tBN+0ymRKRfeno2n4hIs6TrZ+m7C5MpEekXu/lERDII\nYAJKbkymRKRfeq5MXS4XLBaLEm0hIuoaDY2Zdvg4aUVFBcaOHYukpCS8//773ve5WDQRaYaQ/D8U\n1mFlum7dOrz77ruQJAnz58/H5cuXMWXKFIgbPONKRNQtNFSZdphMg4KCvGv3rVmzBs888wzuvPPO\nTpehIiJSi9DQmGmH3fy+ffuisLAQLS0tsFqt+NOf/oQVK1bg1KlTaraPiKhjHo//h8I6TKYFBQUY\nOHCgtxK98847sXnzZjzxxBOKN4qIyC+S8P9QWIfdfLPZjJSUlHbvRUZGYvHixYo3iojILxrq5vM+\nUyLSLz1MQBERaR4XOiEikgErUyKiwAk3n80nIgocK1MiIhncymOm59zq7Hp5rvWSKnEA9XYMBYAt\n1a+oEueJYXNUiQMAIcYg1WINCr1DtVjVzm9UibNKClYlDgDsbzyiWixZsDIlIgqcYDIlIpIBJ6CI\niGTAypSISAZMpkREgdPS+spMpkSkX6xMiYhkwGRKRBQ44dbpTfutra0wGo3cnZSItEE7ubTjlfYB\n4MSJE5g7dy5ycnLw6aefIjk5GcnJyaisrFSrfUREHRKS8PtQms/KdNmyZZg/fz4aGhowb9487N27\nF8HBwXj22WcxduxYxRtHROSTXsZMJUnCyJEjAQCfffYZ+vTpc+VDZg61EpEG6KWbHxcXh8WLF0OS\nJBQVFQEANmzYgMjISFUaR0Tki266+fn5+aioqIDR+J+cGxUVhYyMDMUbRkTUGeG++SQpSRLy8vJQ\nV1cHi8WC/Px8xMbGes9/+eWXKCoqghACt912G1atWoXg4I5X8PJZmRqNRowfP77de5MnT0ZoaOhN\nfwEiItlIXTiuUV5eDpfLhdLSUmRlZXl738CVJ6uWLl2KwsJCvP3220hMTERDQ4PPpnDwk4h0K5C1\noaurq5GYmAgASEhIQG1trfdcfX09evXqhU2bNuHrr7/GI488ggEDBvi8ns/KlIhI0wKoTJ1OJ6xW\nq/e1yWSC2+0GADQ1NeHIkSN4+umn8de//hUHDx7EP/7xD59NYTIlIt0Skv/HtaxWK5qbm72vJUny\n3qnUq1cvxMbGIj4+HkFBQUhMTGxXud4IkykR6ZZw+39ca/jw4fj4448BAA6HAzabzXsuOjoazc3N\nOH36NADg8OHDuPvuu322hWOmRKRbgYyZJiUloaqqCmlpaRBCoKCgAGVlZWhpaYHdbsfLL7+MrKws\nCCEwbNgwjBkzxuf1DELmBQGjIu6R83Id6ttDvXtde5pCVItlNphUifPBkbWqxAGAeNtk1WIFm9Rb\nN+Kyx6VKHLNRvZrHI9TdBuT0uS8D+nzj2Ef8/tmoyo8CitUZVqZEpF/C0N0t8GIyJSLdCqSbLzcm\nUyLSLSGxMiUiCpjkYTIlIgoYu/lERDJgN5+ISAYa2umZyZSI9IuVKRGRDLQ0AeX3s/nnzp1Tsh1E\nRF0mJIPfh9I6rEzr6+vbvV64cCGKi4sBXNnOhIiouwk9PAE1Y8YMhISE4Pbbb4cQAvX19cjNzYXB\nYMDmzZvVbCMR0Q3p4taoHTt2YNmyZUhPT8fo0aORkZGBkpISNdtGROSTpIfKtE+fPnjttddQXFyM\nmpoaNdtEROQXLXXzfU5Amc1mLF682NvVJyLSEslj8PtQml+3RqWkpCAlJUXpthARdQnvMyUikoEu\nxkyJiLROS2OmTKZEpFtamsphMiUi3WI3n4hIBtKtPAF1R2hvuS95Q8HGIFXiAECj66JqsUJU+l5q\n7hh68qv3VIt1/72ZqsX68s/pqsRpKvo/VeIAwJivL6kWSw6sTImIZMAJKCIiGbAyJSKSgYYm85lM\niUi/PJLfSzIrjsmUiHRLQyvwMZkSkX4JcMyUiChgkoYGTZlMiUi3JA1Vpn6P3kqShMbGRkiSlkYp\niOjnTMDg96E0n8l00aJFAICjR49iwoQJePHFF/Hkk0/C4XAo3jAios54YPD7UJrPbv6ZM2cAAK++\n+io2btyI/v37o7GxEVlZWdiyZYvijSMi8kVL/WS/xkxNJhP69+8PAIiKimJXn4g0IZBMJEkS8vLy\nUFdXB4vFgvz8fMTGxl73c0uXLkVERAReeukln9fz2c13Op1ISUlBQ0MDtm3bhsuXL2P58uW46667\nAvgKRETyCGTMtLy8HC6XC6WlpcjKykJRUdF1P7N161Z89dVXfrXFZ2W6c+dOuFwuHD9+HCEhITAY\nDLDZbJg6daqfX5WISDmBrMBXXV2NxMREAEBCQgJqa2vbnf/iiy9w9OhR2O12nDp1qtPrdTqbb7FY\n8Mtf/hI2mw0WiwXp6ekIClJv+Tsioo5IMPh9XMvpdMJqtXpfm0wmuN1uAMDZs2fxxhtvIDc31++2\n8D5TItItTwCftVqtaG5u9r6WJAlm85WUuGfPHjQ1NeG5557Djz/+iNbWVgwYMMDnLs1MpkSkW5Lh\n5vv5w4cPR2VlJZKTk+FwOGCz2bznMjMzkZl5ZaHxnTt34tSpU51ud89kSkS6FcjTpElJSaiqqkJa\nWhqEECgoKEBZWRlaWlpgt9u7fD0mUyLSrUBujTIajVixYkW79+Lj46/7uc4q0quYTIlItzS0nx6T\nKRHplxqPifpL9mQ6ODhK7kve0IZp6j2FNXTTv1WLNSj0DlXiOD2tqsQB1N0x9LOazarFum/I06rE\nibGos+MvAHwQZVEtlhxYmRIRyUBLD7YzmRKRbmlobWgmUyLSL3bziYhkwG4+EZEMPKxMiYgCx8qU\niEgGTKZERDLQ0my+37uTAsD58+chhJaaT0Q/Z5LB/0NpPivTHTt24IcffsDYsWORlZWF4OBgtLa2\nYtmyZXjwwQeVbx0RkQ+66ea/9dZbKCkpwZw5c7B27VrExcWhsbERc+fOZTIlom4XyOLQcvOZTIOC\ngtCjRw+EhYUhOjoawJXdSQ0BLMhKRCQX3dy0P27cOMyZMwc2mw2zZ89GYmIiPvnkE4waNUqt9hER\ndUg33fznnnsOn3/+OQ4cOIC77roL586dQ0ZGBsaMGaNS84iIOqal6fBOb40aOXIkRo4cqUZbiIi6\nRNJQOuV9pkSkW7qZgCIi0jLdjJkSEWmZbmbziYi0jGOmREQy0E4qZTIlIh27pcdMow0hcl/yhlJL\nL6gSBwAkod6cYbXzG1XiqLlgzZd/Tlctllo7hgLA4dotqsT5cPAiVeIAwMSzJ1WLBQDHA/y8R0O1\nKStTItKtW7oyJSJSCyegiIhkoJ1UymRKRDrGbj4RkQw4AUVEJAOOmRIRyUA7qZTJlIh0TDeVqdPp\nhNVqVastRERdEsgElCRJyMvLQ11dHSwWC/Lz8xEbG+s9v3v3bvztb3+DyWSCzWZDXl4ejMaON3T2\nudXz6NGjsW3btgCaS0SkHNGFX9cqLy+Hy+VCaWkpsrKyUFRU5D3X2tqK1157DZs3b8bWrVvhdDpR\nWVnpsy0+k+k999yDY8eOITMzE59//vlNfl0iImV4IPw+rlVdXY3ExEQAQEJCAmpra73nLBYLtm7d\nitDQUACA2+1GcHCwz7b47OYHBwcjNzcXNTU12LBhA1auXIlRo0YhOjoamZmZXf7iRERyCqSbf+0w\npslkgtvthtlshtFoRGRkJACgpKQELS0tGD16tM/r+UymVxfDuPfee7F69WpcunQJhw4dQn19fQBf\ngYhIHlIAC/ZYrVY0Nzf/51qSBLPZ3O71qlWrUF9fj9WrV3e6xb3PZJqSktLudXh4OMaNG3cz7SYi\nkl0gc/nDhw9HZWUlkpOT4XA4YLPZ2p3Pzc2FxWLBmjVrfE48XeUzmU6ZMiWAphIRKSuQW6OSkpJQ\nVVWFtLQ0CCFQUFCAsrIytLS0YMiQIdi+fTvuu+8+PPPMMwCAzMxMJCUldXg93mdKRLp1o1l6fxmN\nRqxYsaLde/Hx8d7fHz/etdVWmUyJSLfcerlpn4hIywKpTOXGZEpEusUl+IiIZKDmXmadkT2ZXoI6\nm899e/mcKnEAoCwiWrVYqyTfT1nI5RPnKVXiAEBT0f+pFivG0lu1WGptdPfoPwtUiQMA0qD/US2W\nHHSz0AkRkZZxcWgiIhmwMiUiksEtPWZKRKQWzuYTEcmA95kSEcmAY6ZERDLwCO109JlMiUi32M0n\nIpJBIItDy61LydTlckGSJISEhCjVHiIiv2knlXayoV59fT3mzZuHrKwsOBwOTJo0CRMnTsT777+v\nVvuIiDokQfh9KM1nZbp06VLMnTsXly5dwuzZs7Fr1y6Eh4djxowZSE5OVrxxRES+6GY23+1248EH\nH4QQAq+88gqioqKufMjMoVYi6n66mc3v27cvFixYAI/Hg7CwMLz66quwWq247bbb1GofEVGHdDOb\nX1xcjI8++gj9+/dHWFgYNm3ahJCQEBQUqLckGBFRR3TzbL7ZbMajjz7qfZ2dna14g4iI/KWbMVMi\nIi3TTWVKRKRlHg2tG8VkSkS6pdsnoIiItEQ3s/lERFp2S1emuy8ek/uSN1S3c4EqcQBgUkaparH2\nNx5RJU7f8D6qxAGAMV9fUi3WB1EW1WJNPHtSlThq7hj6z2PvqBZLDqxMiYhkcEtXpkREatHN46RE\nRFrGbj4RkQwEK1MiosDxcVIiIhlo6XFSnyvtExFpWSAr7UuShNzcXNjtdmRkZOD06dPtzldUVCA1\nNRV2ux3vvNP5LWOsTIlItzzSzY+ZlpeXw+VyobS0FA6HA0VFRVi7di0AoK2tDYWFhdi+fTtCQ0OR\nnp6OcePGITIyssPr+V2ZaqmcJiICrszm+/vrWtXV1UhMTAQAJCQkoLa21nvu5MmTiImJQUREBCwW\nC0aMGIFDhw75bIvPyvTbb7/F8uXLcerUKZw9exaDBw9GdHQ0srOzudo+EXW7QIo8p9MJq9XqfW0y\nmeB2u2E2m+F0OhEeHu49FxYWBqfT6fN6PivT5cuXY8mSJaisrMSbb76J+++/HzNmzMDixYtv+gsQ\nEcklkDFTq9WK5ubm/1xLkrz72117rrm5uV1yvRGfydTpdCIuLg7AlTL4iy++wJAhQ3Dx4kX/vy0R\nkUKEEH4f1xo+fDg+/vhjAIDD4YDNZvOei4+Px+nTp3HhwgW4XC4cPnwYw4YN89kWn938fv36ITc3\nFw8//DD279+PIUOGYP/+/QgNDb2Z701EJKtAJqCSkpJQVVWFtLQ0CCFQUFCAsrIytLS0wG63Izs7\nG7NmzYIQAqmpqd7dmTtiED4GHVwuF7Zt24YTJ05g0KBBSE1NRU1NDWJjY/GLX/zihp+J6X3vTX+5\nrrh1V42q7fyHZKDmqlHBRvVWcvogSr3vNfHseVXiqLmYh9qrRgVFDgjo8xHWeL9/9t9OZVf58lmZ\nWiwWPPXUU+3eS0hIULRBRET+0tJdRrzPlIh0i0vwERHJgKtGERHJgJUpEZEMJC7BR0QUOE5AERHJ\nQEvJ1Od9pkRE5B+uZ0pEJAMmUyIiGTCZEhHJgMmUiEgG3TqbL0kS8vLyUFdXB4vFgvz8fMTGxioW\n7+jRo/jDH/6AkpISxWK0tbVh0aJFaGhogMvlwpw5c/Doo48qEsvj8WDJkiWor6+HwWDA8uXL2y0j\npoRz584hJSUFf/nLXxAf7/8iE101ZcoU78K9/fr1Q2FhoSJx1q9fj4qKCrS1tSE9PR3Tpk1TJM7O\nnTvx97//HQBw+fJlHDt2DFVVVejZs6fssdra2pCdnY2GhgYYjUasXLlSsT8rl8uFnJwcfPfdd7Ba\nrcjNzUX//v0ViaV5ohvt3btXLFy4UAghxJEjR8Tzzz+vWKwNGzaIJ598UkybNk2xGEIIsX37dpGf\nny+EEKKpqUk88sgjisXat2+fyM7OFkIIcfDgQUX/+wkhhMvlEnPnzhWPPfaYOHHihGJxWltbxeTJ\nkxW7/lUHDx4Us2fPFh6PRzidTvH6668rHlMIIfLy8sTWrVsVu/6+ffvEvHnzhBBCHDhwQLz44ouK\nxSopKRFLliwRQghx8uRJMXPmTMViaV23dvN97cEit5iYGKxevVqx61/1+OOPY/78+QCu3ANnMpkU\nizV+/HisXLkSAPD9998rUuX8t+LiYqSlpeH2229XNM7x48fx008/YebMmcjMzITD4VAkzoEDB2Cz\n2fDCCy/g+eefx5gxYxSJ899qampw4sQJ2O12xWLExcXB4/FAkiQ4nU7v6vFKOHHiBB5++GEAwIAB\nA3DypLLL3GlZt3bzfe3BIrcJEybgzJkzsl/3WmFhYQCufLd58+bht7/9raLxzGYzFi5ciH379uH1\n119XLM7OnTvRu3dvJCYmYsOGDYrFAYCQkBDMmjUL06ZNwzfffIPf/OY32LNnj+x/L5qamvD9999j\n3bp1OHPmDObMmYM9e/bAYDDIGue/rV+/Hi+88IJi1weAHj16oKGhAU888QSampqwbt06xWINGjQI\nlZWVGD9+PI4ePYrGxkZ4PB5Fiwit6tbK1NceLHr2ww8/IDMzE5MnT8akSZMUj1dcXIy9e/di6dKl\naGlpUSTGjh078OmnnyIjIwPHjh3DwoUL8eOPPyoSKy4uDr/61a9gMBgQFxeHXr16KRKrV69eeOih\nh2CxWDBgwAAEBwfj/HnlFny+ePEi6uvrMWrUKMViAMCmTZvw0EMPYe/evXjvvfeQnZ2Ny5cvKxIr\nNTUVVqsV06dPx759+zB48OCfZSIFujmZ+tqDRa/+9a9/YebMmfjd736HqVOnKhrr3Xffxfr16wEA\noaGhMBgMMBqV+SN98803sWXLFpSUlGDQoEEoLi5WbIfa7du3o6ioCADQ2NgIp9OpSKwRI0bgk08+\ngRACjY2N+Omnn9CrVy/Z41x16NAhPPDAA4pd/6qePXt6N3+LiIiA2+2Gx+NRJFZNTQ0eeOABvP32\n23j88ccRHR2tSBw96NYy8EZ7sOjdunXrcPHiRaxZswZr1qwBAGzcuBEhISGyx3rssceQk5ODp556\nCm63G4sWLVIkjtqmTp2KnJwcpKenw2AwoKCgQJEey9ixY3Ho0CFMnToVQgjk5uYqWlXV19ejX79+\nil3/ql//+tdYtGgRpk+fjra2NixYsAA9evRQJFZsbCz++Mc/Yt26dQgPD8fLL7+sSBw94LP5REQy\n4E37REQyYDIlIpIBkykRkQyYTImIZMBkSkQkAyZTIiIZMJkSEcng/wGrqD3VEndeiwAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a20ffec18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Let's compute a similarity matrix from the tfidf LSA analysis.\n",
    "similarity = np.asarray(np.asmatrix(X_train_lsa) * np.asmatrix(X_train_lsa).T)\n",
    "sim_mat = pd.DataFrame(similarity).iloc[0:10, 0:10]\n",
    "sim_full = pd.DataFrame(similarity)\n",
    "sns.heatmap(sim_mat, yticklabels=range(sim_mat.shape[1]))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:45:10.975053Z",
     "start_time": "2018-03-02T19:45:10.967172Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "indonesia sees cpo price rising sharply indonesia expects crude palm oil (cpo) prices to rise sharply to between 450 and 550 dlrs a tonne fob sometime this year because of better european demand and a fall in malaysian output, hasrul harahap, junior minister for tree crops, told indonesian reporters \n",
      "\n",
      " malaysia may not meet 1987 oil palm target malaysia is unlikely to meet its targeted output of five mln tonnes of oil palm in calendar 1987, oil palm growers told reuters. output in 1987 is expected to reach around 4.5 mln tonnes, unchanged from 1986, because of drought, low use of fertiliser and ov\n"
     ]
    }
   ],
   "source": [
    "# Articles 1 and 9 should be fairly similar to each other.\n",
    "print(articles_train[1][:300], '\\n\\n', articles_train[9][:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both articles are about palm oil. Very cool. Let's look at some of the BEST matches in the whole data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:45:24.426476Z",
     "start_time": "2018-03-02T19:45:24.386769Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    463\n",
      "1     27\n",
      "2    151\n",
      "3    702\n",
      "4    319\n",
      "5    584\n",
      "6    138\n",
      "7    635\n",
      "8    390\n",
      "9    245\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Remove the 1's from the diagonal and find the remaining max index.\n",
    "print((sim_full - np.identity(len(sim_full))).idxmax()[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-02T19:45:55.158274Z",
     "start_time": "2018-03-02T19:45:55.146100Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity: 0.998731024989\n",
      "\n",
      " bank of japan satisfied with yen at current range the bank of japan is satisfied with the yen around its current range, a senior central bank official told reporters. he said the pledge by major industrial nations in paris last month to cooperate to hold exchange rates around current ranges applied  \n",
      "\n",
      " bank of japan satisfied with yen at current range the bank of japan is satisfied with the yen around its current range, a senior central bank official told reporters. he said the pledge by major industrial nations in paris last month to cooperate to hold exchange rates around current ranges applied \n"
     ]
    }
   ],
   "source": [
    "items = [0, 463]\n",
    "print('Similarity:', sim_full.iloc[items[0], items[1]])\n",
    "print('\\n', list(articles_train)[items[0]][:300], '\\n\\n', list(articles_train)[items[1]][:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-02-28T21:03:15.725118Z",
     "start_time": "2018-02-28T21:03:15.713348Z"
    },
    "collapsed": true
   },
   "source": [
    "So, this is actually almost an exact duplicate of the same article. The second example simply has a couple extra sentences added at the end, which I've cropped for display purposes. No wonder they have such a high similarity score! I wonder how many near-duplicates were in our data set. In the future, this would be a great thing to check at the very beginning of a project..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Conclusion\n",
    "\n",
    "Multilabel data is common in the 'real world', and we've been able to do quite well categorizing Reuters articles using tfidf with latent semantic analysis and KMeans. With KMeans, we were able to go from a score of 0.54 on the Bag of Words test set with our custom evaluation metric to 0.83 on the tfidf test set. Our model used 32 clusters, and our original data included 15 multilabel categories. Latent semantic analysis did well separating articles by category, and also separated articles by commonalities other than the given categories. \n",
    "\n",
    "Tfidf proved its value by substantially outperforming the Bag of Words method. It was particularly clear that correlation between common words was a sticking point in our BoW analysis, and tfidf resolved that. This kind of analysis has broad applicability in categorization and tagging of any corpus of articles, memos, or other text documents. Although our articles already had tags we could use to categorize the clusters we created, it would be a relatively straight-forward task to create those labels automatically based on the features used to make the cluster decisions. A project for another day!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
